from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	2.299420	0.185296	NaN	...	brute	-1	1	0.663	0.352257	0.017188	1.000	6.527671	6.535437	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.028985	0.003397	NaN	...	brute	-1	1	1.000	18.966825	0.284876	0.757	0.001528	0.001528	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	3.125257	0.066843	NaN	...	brute	-1	5	0.757	18.798363	0.024306	0.882	0.166252	0.166252	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.444856	0.014145	NaN	...	brute	1	100	0.882	0.369740	0.017971	1.000	6.612358	6.620164	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.025754	0.001862	NaN	...	brute	1	100	1.000	19.015915	0.134237	0.757	0.001354	0.001354	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	3.145997	0.066497	NaN	...	brute	-1	100	0.882	18.845010	0.012034	0.663	0.166941	0.166941	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.457043	0.038789	NaN	...	brute	1	5	0.757	0.273782	0.009487	1.000	8.974445	8.979832	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.024767	0.001209	NaN	...	brute	1	5	1.000	4.301663	0.033928	0.922	0.005758	0.005758	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.616419	0.026886	NaN	...	brute	1	1	0.663	4.378157	0.031494	0.929	0.369201	0.369210	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.822141	0.023534	NaN	...	brute	-1	1	0.896	0.276001	0.013155	1.000	6.601936	6.609430	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.006277	0.004084	NaN	...	brute	-1	1	1.000	4.286233	0.049342	0.922	0.001464	0.001464	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.731999	0.022395	NaN	...	brute	-1	5	0.922	4.284738	0.037053	0.896	0.637612	0.637635	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.674	0.0	-1	1	19.262	0.220	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.768	0.0	-1	5	0.347	0.015	1.000	0.040	0.040	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.912	0.0	1	100	19.095	0.024	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.843	0.0	-1	100	0.356	0.012	1.000	0.038	0.038	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.015	0.001	5.332	0.0	1	5	4.317	0.036	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.761	0.0	1	1	0.274	0.009	1.000	0.051	0.051	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.344	0.0	-1	1	4.384	0.039	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.310	0.0	-1	5	0.287	0.013	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.299	0.185	0.0	0.002	-1	1	0.352	0.017	1.000	6.528	6.535	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.029	0.003	0.0	0.029	-1	1	18.967	0.285	0.757	0.002	0.002	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.125	0.067	0.0	0.003	-1	5	18.798	0.024	0.882	0.166	0.166	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.030	0.004	0.0	0.030	-1	5	0.377	0.008	1.000	0.081	0.081	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.445	0.014	0.0	0.002	1	100	0.370	0.018	1.000	6.612	6.620	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.002	0.0	0.026	1	100	19.016	0.134	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.146	0.066	0.0	0.003	-1	100	18.845	0.012	0.663	0.167	0.167	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.033	0.004	0.0	0.033	-1	100	0.375	0.016	1.000	0.087	0.087	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.457	0.039	0.0	0.002	1	5	0.274	0.009	1.000	8.974	8.980	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.001	0.0	0.025	1	5	4.302	0.034	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.616	0.027	0.0	0.002	1	1	4.378	0.031	0.929	0.369	0.369	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.001	0.0	0.024	1	1	0.294	0.011	1.000	0.083	0.083	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.822	0.024	0.0	0.002	-1	1	0.276	0.013	1.000	6.602	6.609	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	0.0	0.006	-1	1	4.286	0.049	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.732	0.022	0.0	0.003	-1	5	4.285	0.037	0.896	0.638	0.638	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.010	0.003	0.0	0.010	-1	5	0.272	0.012	1.000	0.038	0.038	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.877259	1.077265	NaN	...	kd_tree	-1	1	0.929	3.054793	0.258000	1.000	0.287175	0.288197	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.003201	0.000538	NaN	...	kd_tree	-1	1	1.000	131.140779	0.000000	0.946	0.000024	0.000024	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.134279	0.412869	NaN	...	kd_tree	-1	5	0.946	136.291202	0.000000	0.951	0.008322	0.008322	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	6.121749	0.632406	NaN	...	kd_tree	1	100	0.951	3.170403	0.191043	1.000	1.930906	1.934408	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003393	0.000785	NaN	...	kd_tree	1	100	1.000	131.942419	0.000000	0.946	0.000026	0.000026	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.530933	0.233538	NaN	...	kd_tree	-1	100	0.951	131.171652	0.000000	0.929	0.026918	0.026918	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.854643	0.395971	NaN	...	kd_tree	1	5	0.946	0.006323	0.000232	1.000	293.331897	293.528647	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001793	0.000347	NaN	...	kd_tree	1	5	1.000	0.045270	0.001970	0.911	0.039599	0.039636	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	1.013470	0.380790	NaN	...	kd_tree	1	1	0.929	0.070244	0.003254	0.894	14.427824	14.443297	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.033248	0.013085	NaN	...	kd_tree	-1	1	0.891	0.005929	0.000291	1.000	5.607367	5.614095	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.003427	0.000615	NaN	...	kd_tree	-1	1	1.000	0.046870	0.002363	0.911	0.073116	0.073209	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.030908	0.001397	NaN	...	kd_tree	-1	5	0.911	0.045419	0.001583	0.891	0.680506	0.680919	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.898	0.065	0.028	0.0	-1	1	129.797	0.000	0.929	0.022	0.022	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.877	0.053	0.021	0.0	-1	5	3.151	0.251	1.000	1.231	1.235	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.987	0.078	0.020	0.0	1	100	133.368	0.000	0.951	0.030	0.030	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.967	0.107	0.020	0.0	-1	100	3.148	0.175	1.000	1.260	1.262	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.004	0.054	0.020	0.0	1	5	0.048	0.002	0.891	83.525	83.585	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.958	0.068	0.020	0.0	1	1	0.007	0.000	1.000	600.951	601.836	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.019	0.0	-1	1	0.073	0.003	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.020	0.0	-1	5	0.006	0.000	1.000	0.136	0.136	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.877	1.077	0.000	0.001	-1	1	3.055	0.258	1.000	0.287	0.288	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	131.141	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.134	0.413	0.000	0.001	-1	5	136.291	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	-1	5	3.237	0.203	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	6.122	0.632	0.000	0.006	1	100	3.170	0.191	1.000	1.931	1.934	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	131.942	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.531	0.234	0.000	0.004	-1	100	131.172	0.000	0.929	0.027	0.027	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	3.223	0.226	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.855	0.396	0.000	0.002	1	5	0.006	0.000	1.000	293.332	293.529	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.045	0.002	0.911	0.040	0.040	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.013	0.381	0.000	0.001	1	1	0.070	0.003	0.894	14.428	14.443	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.006	0.000	1.000	0.172	0.172	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.033	0.013	0.000	0.000	-1	1	0.006	0.000	1.000	5.607	5.614	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	0.000	0.003	-1	1	0.047	0.002	0.911	0.073	0.073	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.031	0.001	0.001	0.000	-1	5	0.045	0.002	0.891	0.681	0.681	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	5	0.006	0.001	1.000	0.424	0.427	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.18818	0.009651	300	...	100	100	10	0.824	0.52256	0.026759	1.0	0.360111	0.360583	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	124.918	0.0	300	0.001	0.001	0.629	0.046	0.824	198.583	199.108	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.188	0.01	300	0.004	0.0	0.523	0.027	1.0	0.36	0.361	See	See